tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

Enselic · 2025-06-25T06:10:21Z

And introduce two new directives for ui tests:

run-crash
run-fail-or-crash

Normally a run-fail ui test like tests that panic shall not be terminated by a signal like SIGABRT. So begin having that as a hard requirement.

Some of our current tests do terminate by a signal/crash however. Introduce and use run-crash for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: 0xc000001d (STATUS_ILLEGAL_INSTRUCTION). Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127".

Some tests behave differently on different targets:

Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for run-fail tests for such targets.
Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use run-fail-or-crash for such tests.

This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as SIGABRT in tests/ui/panics/panic-main.rs (shown as Aborted (core dumped) in the logs attached to that issue, and I have also been able to reproduce this locally).

TODO

Q: what about on Windows?
A: we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
test all permutations of actual vs expected
Done: See tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002 (comment).
Handle targets without unwind support
Add run-fail-or-crash for some sanitizer tests
remote-test-client. See remote-test-client: Exit code 128 + <signal-number> instead of 3 #143448

Zulip discussion

See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-gnu
try-job: dist-i586-gnu-i586-i686-musl
try-job: test-various
try-job: armhf-gnu

rustbot · 2025-06-25T06:10:25Z

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-06-25T06:10:29Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

workingjubilee · 2025-06-25T08:18:58Z

why not simply run-abort

also would accept run-crash

jieyouxu · 2025-06-25T10:10:37Z

@bors2 delegate=try

rust-bors · 2025-06-25T10:10:40Z

@Enselic can now perform try builds on this pull request

Enselic · 2025-06-25T11:37:25Z

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T11:37:29Z

⌛ Trying commit 17be091 with merge 873ecba…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] what about on Windows? - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

Enselic · 2025-06-25T13:34:03Z

An only-windows test that was run-fail now must be run-crash. Fixed now. Trying again:

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T13:34:07Z

⌛ Trying commit ad4e082 with merge e685982…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code - [ ] test all permutations of actual vs expected ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

…, r=Mark-Simulacrum remote-test-client: Exit code `128 + <signal-number>` instead of `3` If the remote process is terminated by a signal, make `remote-test-client` exit with the code `128 + <signal-number>` instead of always `3`. This follows common practice among tools such as bash [^1]: > When a command terminates on a fatal signal whose number is N, Bash uses the > value 128+N as the exit status. It also allows us to differentiate between `run-pass` and `run-crash` ui tests without special case code in compiletest for that when `remote-test-client` is used. See rust-lang#143002 and in particular rust-lang#143002 (comment). Exiting with code `3` has been done from the start (see rust-lang#39400) and seems arbitrary rather than a deliberate design decision, so changing it does not seem like an extraordinarily big deal. ### Regression testing Note that rust-lang#143002 will act as a regression test once it is rebased on this PR. ### Why a separate PR I think it is comforting to know that CI does not break with just this change. But if my reviewer prefers, we can move this commit to be part of rust-lang#143002 instead. [^1]: https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html

Rollup merge of #143448 - Enselic:remote-test-client-signals, r=Mark-Simulacrum remote-test-client: Exit code `128 + <signal-number>` instead of `3` If the remote process is terminated by a signal, make `remote-test-client` exit with the code `128 + <signal-number>` instead of always `3`. This follows common practice among tools such as bash [^1]: > When a command terminates on a fatal signal whose number is N, Bash uses the > value 128+N as the exit status. It also allows us to differentiate between `run-pass` and `run-crash` ui tests without special case code in compiletest for that when `remote-test-client` is used. See #143002 and in particular #143002 (comment). Exiting with code `3` has been done from the start (see #39400) and seems arbitrary rather than a deliberate design decision, so changing it does not seem like an extraordinarily big deal. ### Regression testing Note that #143002 will act as a regression test once it is rebased on this PR. ### Why a separate PR I think it is comforting to know that CI does not break with just this change. But if my reviewer prefers, we can move this commit to be part of #143002 instead. [^1]: https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html

…t ok) And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d`. Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests.

Enselic · 2025-07-16T19:32:44Z

I have now rebased on #143448 and the rest of the changes on master, so the changes from the last approved commit 6950daf are basically just cosmetic (see diff of diff below). I also had to change from run-fail to run-crash for three new tests.

So I will r=@petrochenkov this again after PR CI passes.

Click to expand diff of diff (yes it's a bit confusing but illustrates that the changes are minor)

diff -u <(git show 6950daf) <(git show f120fbe)

--- /proc/self/fd/15	2025-07-16 21:26:23.161561664 +0200
+++ /proc/self/fd/20	2025-07-16 21:26:23.161561664 +0200
@@ -1,4 +1,4 @@
-commit 6950daf354e1f9718583578e71cdad3cf16360cb (origin/tests-ui-run-fail-exit-vs-signal-20)
+commit f120fbe5c1b42dd08233ecf64eb45bc937319412 (HEAD -> tests-ui-run-fail-exit-vs-signal, origin/tests-ui-run-fail-exit-vs-signal-21, origin/tests-ui-run-fail-exit-vs-signal)
 Author: Martin Nordholts <[email protected]>
 Date:   Wed Jun 25 07:56:40 2025 +0200
 
@@ -45,7 +45,7 @@
  | `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental`                       | N/A             |
  | `failure-status`            | Check                                       | `ui`, `crashes`                           | Any `u16`       |
 diff --git a/src/doc/rustc-dev-guide/src/tests/ui.md b/src/doc/rustc-dev-guide/src/tests/ui.md
-index f7e62e1eccf..7353e941e8e 100644
+index 4fce5838b6e..9bfc60e08a6 100644
 --- a/src/doc/rustc-dev-guide/src/tests/ui.md
 +++ b/src/doc/rustc-dev-guide/src/tests/ui.md
 @@ -448,7 +448,7 @@ even run the resulting program. Just add one of the following
@@ -58,8 +58,8 @@
    - `//@ check-fail` — compilation should fail (the codegen phase is skipped).
      This is the default for UI tests.
 @@ -457,10 +457,20 @@ even run the resulting program. Just add one of the following
-     without the codegen phase, then a second time the full compile should
-     fail.
+     - First time is to ensure that the compile succeeds without the codegen phase
+     - Second time is to ensure that the full compile fails
    - `//@ run-fail` — compilation should succeed, but running the resulting
 -    binary should fail.
 -
@@ -83,10 +83,10 @@
  If you want to check the output of running the program, include the
  `check-run-results` directive. This will check for a `.run.stderr` and
 diff --git a/src/tools/compiletest/src/common.rs b/src/tools/compiletest/src/common.rs
-index cdce5d941d0..21b5217d49b 100644
+index 33da1a25db1..c83070aaba7 100644
 --- a/src/tools/compiletest/src/common.rs
 +++ b/src/tools/compiletest/src/common.rs
-@@ -111,11 +111,37 @@ pub enum PassMode {
+@@ -88,11 +88,37 @@ pub enum PassMode {
      }
  }
  
@@ -125,34 +125,20 @@
  }
  
  string_enum! {
-diff --git a/src/tools/compiletest/src/directive-list.rs b/src/tools/compiletest/src/directive-list.rs
-index adf2a7bffef..84d162043fe 100644
---- a/src/tools/compiletest/src/directive-list.rs
-+++ b/src/tools/compiletest/src/directive-list.rs
-@@ -241,7 +241,9 @@
-     "regex-error-pattern",
-     "remap-src-base",
-     "revisions",
-+    "run-crash",
-     "run-fail",
-+    "run-fail-or-crash",
-     "run-flags",
-     "run-pass",
-     "run-rustfix",
 diff --git a/src/tools/compiletest/src/directives.rs b/src/tools/compiletest/src/directives.rs
-index a6242cf0c22..e10eb8ab39e 100644
+index 93133ea0bfd..513716357f4 100644
 --- a/src/tools/compiletest/src/directives.rs
 +++ b/src/tools/compiletest/src/directives.rs
 @@ -9,7 +9,7 @@
  use semver::Version;
  use tracing::*;
  
--use crate::common::{Config, Debugger, FailMode, Mode, PassMode};
-+use crate::common::{Config, Debugger, FailMode, Mode, PassMode, RunFailMode};
+-use crate::common::{Config, Debugger, FailMode, PassMode, TestMode};
++use crate::common::{Config, Debugger, FailMode, PassMode, RunFailMode, TestMode};
  use crate::debuggers::{extract_cdb_version, extract_gdb_version};
  use crate::directives::auxiliary::{AuxProps, parse_and_update_aux};
  use crate::directives::needs::CachedNeedsConditions;
-@@ -656,7 +656,13 @@ fn update_fail_mode(&mut self, ln: &str, config: &Config) {
+@@ -654,7 +654,13 @@ fn update_fail_mode(&mut self, ln: &str, config: &Config) {
              Some(FailMode::Build)
          } else if config.parse_name_directive(ln, "run-fail") {
              check_ui("run");
@@ -167,38 +153,41 @@
          } else {
              None
          };
+@@ -1007,7 +1013,9 @@ fn line_directive<'line>(
+     "regex-error-pattern",
+     "remap-src-base",
+     "revisions",
++    "run-crash",
+     "run-fail",
++    "run-fail-or-crash",
+     "run-flags",
+     "run-pass",
+     "run-rustfix",
 diff --git a/src/tools/compiletest/src/runtest.rs b/src/tools/compiletest/src/runtest.rs
-index f8bf4ee3022..6833b27253a 100644
+index cb8f593c9df..f66d4f98f1f 100644
 --- a/src/tools/compiletest/src/runtest.rs
 +++ b/src/tools/compiletest/src/runtest.rs
-@@ -17,10 +17,10 @@
+@@ -16,8 +16,8 @@
+ use tracing::*;
  
  use crate::common::{
-     Assembly, Codegen, CodegenUnits, CompareMode, Config, CoverageMap, CoverageRun, Crashes,
--    DebugInfo, Debugger, FailMode, Incremental, MirOpt, PassMode, Pretty, RunMake, Rustdoc,
--    RustdocJs, RustdocJson, TestPaths, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT,
--    UI_STDERR, UI_STDOUT, UI_SVG, UI_WINDOWS_SVG, Ui, expected_output_path, incremental_dir,
--    output_base_dir, output_base_name, output_testname_unique,
-+    DebugInfo, Debugger, FailMode, Incremental, MirOpt, PassMode, Pretty, RunFailMode, RunMake,
-+    RunResult, Rustdoc, RustdocJs, RustdocJson, TestPaths, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR,
-+    UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG, UI_WINDOWS_SVG, Ui, expected_output_path,
-+    incremental_dir, output_base_dir, output_base_name, output_testname_unique,
+-    CompareMode, Config, Debugger, FailMode, PassMode, TestMode, TestPaths, TestSuite,
+-    UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG,
++    CompareMode, Config, Debugger, FailMode, PassMode, RunFailMode, RunResult, TestMode, TestPaths,
++    TestSuite, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG,
+     UI_WINDOWS_SVG, expected_output_path, incremental_dir, output_base_dir, output_base_name,
+     output_testname_unique,
  };
- use crate::compute_diff::{DiffLine, make_diff, write_diff, write_filtered_diff};
- use crate::directives::TestProps;
-@@ -277,7 +277,11 @@ fn pass_mode(&self) -> Option<PassMode> {
- 
+@@ -282,7 +282,8 @@ fn pass_mode(&self) -> Option<PassMode> {
      fn should_run(&self, pm: Option<PassMode>) -> WillExecute {
          let test_should_run = match self.config.mode {
--            Ui if pm == Some(PassMode::Run) || self.props.fail_mode == Some(FailMode::Run) => true,
-+            Ui if pm == Some(PassMode::Run)
-+                || matches!(self.props.fail_mode, Some(FailMode::Run(_))) =>
-+            {
-+                true
-+            }
-             MirOpt if pm == Some(PassMode::Run) => true,
-             Ui | MirOpt => false,
-             mode => panic!("unimplemented for mode {:?}", mode),
+             TestMode::Ui
+-                if pm == Some(PassMode::Run) || self.props.fail_mode == Some(FailMode::Run) =>
++                if pm == Some(PassMode::Run)
++                    || matches!(self.props.fail_mode, Some(FailMode::Run(_))) =>
+             {
+                 true
+             }
 diff --git a/src/tools/compiletest/src/runtest/ui.rs b/src/tools/compiletest/src/runtest/ui.rs
 index f6bc85cd051..0507c2600ae 100644
 --- a/src/tools/compiletest/src/runtest/ui.rs
@@ -489,20 +478,20 @@
  //@ ignore-i686-pc-windows-msvc: #112480
  //@ compile-flags: -C debug-assertions
  //@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is
-diff --git a/tests/ui/mir/enum/convert_non_enum_break.rs b/tests/ui/mir/enum/convert_non_enum_break.rs
-index de062c39907..366f48dd77c 100644
---- a/tests/ui/mir/enum/convert_non_enum_break.rs
-+++ b/tests/ui/mir/enum/convert_non_enum_break.rs
+diff --git a/tests/ui/mir/enum/convert_non_integer_break.rs b/tests/ui/mir/enum/convert_non_integer_break.rs
+index 29795190bf6..b0778e2024f 100644
+--- a/tests/ui/mir/enum/convert_non_integer_break.rs
++++ b/tests/ui/mir/enum/convert_non_integer_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
  //@ compile-flags: -C debug-assertions
- //@ error-pattern: trying to construct an enum from an invalid value 0x10000
+ //@ error-pattern: trying to construct an enum from an invalid value
  
-diff --git a/tests/ui/mir/enum/convert_non_enum_niche_break.rs b/tests/ui/mir/enum/convert_non_enum_niche_break.rs
+diff --git a/tests/ui/mir/enum/convert_non_integer_niche_break.rs b/tests/ui/mir/enum/convert_non_integer_niche_break.rs
 index 9ff4849c5b1..d26a3aeb506 100644
---- a/tests/ui/mir/enum/convert_non_enum_niche_break.rs
-+++ b/tests/ui/mir/enum/convert_non_enum_niche_break.rs
+--- a/tests/ui/mir/enum/convert_non_integer_niche_break.rs
++++ b/tests/ui/mir/enum/convert_non_integer_niche_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
@@ -520,14 +509,14 @@
  //@ error-pattern: trying to construct an enum from an invalid value 0xfd
  
 diff --git a/tests/ui/mir/enum/niche_option_tuple_break.rs b/tests/ui/mir/enum/niche_option_tuple_break.rs
-index 43eef3a4cc5..70d4de1f3d0 100644
+index affdc4784a3..0a933afa153 100644
 --- a/tests/ui/mir/enum/niche_option_tuple_break.rs
 +++ b/tests/ui/mir/enum/niche_option_tuple_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
  //@ compile-flags: -C debug-assertions
- //@ error-pattern: trying to construct an enum from an invalid value 0x3
+ //@ error-pattern: trying to construct an enum from an invalid value
  
 diff --git a/tests/ui/mir/enum/numbered_variants_break.rs b/tests/ui/mir/enum/numbered_variants_break.rs
 index e3e71dc8aec..fbe7d6627a3 100644
@@ -570,14 +559,14 @@
  //@ error-pattern: trying to construct an enum from an invalid value 0x1
  
 diff --git a/tests/ui/mir/enum/with_niche_int_break.rs b/tests/ui/mir/enum/with_niche_int_break.rs
-index 0ec60a33564..9747bb8cfaf 100644
+index 6a97eaa8f4f..d363dc7568a 100644
 --- a/tests/ui/mir/enum/with_niche_int_break.rs
 +++ b/tests/ui/mir/enum/with_niche_int_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
  //@ compile-flags: -C debug-assertions
- //@ error-pattern: trying to construct an enum from an invalid value 0x4
+ //@ error-pattern: trying to construct an enum from an invalid value
  
 diff --git a/tests/ui/mir/enum/wrap_break.rs b/tests/ui/mir/enum/wrap_break.rs
 index 4491394ca5a..5c410afa511 100644
@@ -931,6 +920,36 @@
  //@ compile-flags: -Copt-level=3 -Cdebug-assertions=no -Zub-checks=yes
  //@ error-pattern: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached
  
+diff --git a/tests/ui/precondition-checks/vec-from-parts.rs b/tests/ui/precondition-checks/vec-from-parts.rs
+index 0bafb5aa715..ace90770360 100644
+--- a/tests/ui/precondition-checks/vec-from-parts.rs
++++ b/tests/ui/precondition-checks/vec-from-parts.rs
+@@ -1,4 +1,4 @@
+-//@ run-fail
++//@ run-crash
+ //@ compile-flags: -Cdebug-assertions=yes
+ //@ error-pattern: unsafe precondition(s) violated: Vec::from_parts_in requires that length <= capacity
+ #![feature(allocator_api)]
+diff --git a/tests/ui/precondition-checks/vec-from-raw-parts.rs b/tests/ui/precondition-checks/vec-from-raw-parts.rs
+index 884d34c0a56..1bc8e6ada10 100644
+--- a/tests/ui/precondition-checks/vec-from-raw-parts.rs
++++ b/tests/ui/precondition-checks/vec-from-raw-parts.rs
+@@ -1,4 +1,4 @@
+-//@ run-fail
++//@ run-crash
+ //@ compile-flags: -Cdebug-assertions=yes
+ //@ error-pattern: unsafe precondition(s) violated: Vec::from_raw_parts_in requires that length <= capacity
+ //@ revisions: vec_from_raw_parts vec_from_raw_parts_in string_from_raw_parts
+diff --git a/tests/ui/precondition-checks/vec-set-len.rs b/tests/ui/precondition-checks/vec-set-len.rs
+index 0987e7fe028..c6bdee7dc67 100644
+--- a/tests/ui/precondition-checks/vec-set-len.rs
++++ b/tests/ui/precondition-checks/vec-set-len.rs
+@@ -1,4 +1,4 @@
+-//@ run-fail
++//@ run-crash
+ //@ compile-flags: -Cdebug-assertions=yes
+ //@ error-pattern: unsafe precondition(s) violated: Vec::set_len requires that new_len <= capacity()
+ 
 diff --git a/tests/ui/precondition-checks/write_volatile.rs b/tests/ui/precondition-checks/write_volatile.rs
 index 0d5ecb014b3..25107871c39 100644
 --- a/tests/ui/precondition-checks/write_volatile.rs

Enselic · 2025-07-17T03:29:42Z

(See above)

@bors r=@petrochenkov

bors · 2025-07-17T03:29:45Z

📌 Commit f120fbe has been approved by petrochenkov

It is now in the queue for this repository.

…Simulacrum remote-test-client: Exit code `128 + <signal-number>` instead of `3` If the remote process is terminated by a signal, make `remote-test-client` exit with the code `128 + <signal-number>` instead of always `3`. This follows common practice among tools such as bash [^1]: > When a command terminates on a fatal signal whose number is N, Bash uses the > value 128+N as the exit status. It also allows us to differentiate between `run-pass` and `run-crash` ui tests without special case code in compiletest for that when `remote-test-client` is used. See rust-lang/rust#143002 and in particular rust-lang/rust#143002 (comment). Exiting with code `3` has been done from the start (see rust-lang/rust#39400) and seems arbitrary rather than a deliberate design decision, so changing it does not seem like an extraordinarily big deal. ### Regression testing Note that rust-lang/rust#143002 will act as a regression test once it is rebased on this PR. ### Why a separate PR I think it is comforting to know that CI does not break with just this change. But if my reviewer prefers, we can move this commit to be part of rust-lang/rust#143002 instead. [^1]: https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html

…signal, r=petrochenkov tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) And introduce two new directives for ui tests: * `run-crash` * `run-fail-or-crash` Normally a `run-fail` ui test like tests that panic shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal/crash however. Introduce and use `run-crash` for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: `0xc000001d` (`STATUS_ILLEGAL_INSTRUCTION`). Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127". Some tests behave differently on different targets: * Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for `run-fail` tests for such targets. * Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use `run-fail-or-crash` for such tests. This adds further (cc rust-lang#142304, rust-lang#142886) protection against the regression in rust-lang#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See rust-lang#143002 (comment). - [x] Handle targets without unwind support - [x] Add `run-fail-or-crash` for some sanitizer tests - [x] remote-test-client. See rust-lang#143448 ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various try-job: armhf-gnu

fmease · 2025-07-17T12:58:41Z

I've got a CI failure in a rollup: #144067 (comment). Not sure if it's caused by this PR alone or together with other ones conglomerated. Let's rerun CI and preemptively kick this one out of the queue for now.

@bors r-

rust-log-analyzer · 2025-07-17T13:46:50Z

The job aarch64-gnu-llvm-19-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

failures:

---- [ui] tests/ui/panics/panic-main.rs#abort-full stdout ----

error in revision `abort-full`: test did not exit with failure! code=None so test would pass with `run-crash`
status: signal: 6 (SIGABRT) (core dumped)
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main.abort-full" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="full" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main.abort-full/a"
stdout: none
--- stderr -------------------------------

thread 'main' panicked at /checkout/tests/ui/panics/panic-main.rs:29:5:
moop
---


---- [ui] tests/ui/panics/panic-main.rs#abort-one stdout ----

error in revision `abort-one`: test did not exit with failure! code=None so test would pass with `run-crash`
status: signal: 6 (SIGABRT) (core dumped)
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main.abort-one" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="1" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main.abort-one/a"
stdout: none
--- stderr -------------------------------

thread 'main' panicked at /checkout/tests/ui/panics/panic-main.rs:29:5:
moop
---


---- [ui] tests/ui/panics/panic-main.rs#abort-zero stdout ----

error in revision `abort-zero`: test did not exit with failure! code=None so test would pass with `run-crash`
status: signal: 6 (SIGABRT) (core dumped)
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main.abort-zero" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_BACKTRACE="0" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main.abort-zero/a"
stdout: none
--- stderr -------------------------------

thread 'main' panicked at /checkout/tests/ui/panics/panic-main.rs:29:5:
moop

fmease · 2025-07-17T13:48:12Z

@rustbot author

rustbot assigned petrochenkov Jun 25, 2025

Enselic added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2025

Enselic mentioned this pull request Jun 25, 2025

RUST_BACKTRACE=full loop with -Cpanic=abort on aarch64-unknown-linux-gnu #123733

Closed

rustbot added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Jun 25, 2025

This comment has been minimized.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 6ad1973 to 17be091 Compare June 25, 2025 10:12

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 17be091 to ad4e082 Compare June 25, 2025 13:30

This comment was marked as resolved.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from ad4e082 to 2499dac Compare June 25, 2025 16:29

rustbot added the A-rustc-dev-guide Area: rustc-dev-guide label Jun 25, 2025

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from b5956bf to f120fbe Compare July 16, 2025 19:21

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 17, 2025

fmease mentioned this pull request Jul 17, 2025

Rollup of 16 pull requests #144067

Closed

fmease closed this Jul 17, 2025

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 17, 2025

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jul 17, 2025

fmease reopened this Jul 17, 2025

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 17, 2025

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 17, 2025

tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002

Are you sure you want to change the base?

tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002

Conversation

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Zulip discussion

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

This comment has been minimized.

workingjubilee commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Enselic commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as resolved.

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as resolved.

Enselic commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Enselic commented Jul 17, 2025

Uh oh!

bors commented Jul 17, 2025

Uh oh!

fmease commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-log-analyzer commented Jul 17, 2025

Uh oh!

fmease commented Jul 17, 2025

Uh oh!

Uh oh!

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

Enselic commented Jun 25, 2025 •

edited

Loading

workingjubilee commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 25, 2025 •

edited

Loading

Enselic commented Jul 16, 2025 •

edited

Loading

fmease commented Jul 17, 2025 •

edited

Loading